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ABSTRACT 

Hierarchical image segmentation provides region-oriented scale- 
space, i.e., a set of image segmentations at different detail levels 
in which the segmentations at finer levels are nested with respect 
to those at coarser levels. Most image segmentation algorithms, 
such as region merging algorithms, rely on a criterion for merging 
that does not lead to a hierarchy, and for which the tuning of the 
parameters can be difficult. In this work, we propose a hierarchical 
graph based image segmentation relying on a criterion popularized 
by Felzenzwalb and Huttenlocher. We illustrate with both real and 
synthetic images, showing efficiency, ease of use, and robustness of 
our method. 

Index Terms — Hierarchical image segmentation, Edge- weighted 
graph, Saliency map 

1. INTRODUCTION 

Image segmentation is the process of grouping perceptually simi- 
lar pixels into regions. A hierarchical image segmentation is a set of 
image segmentations at different detail levels in which the segmenta- 
tions at coarser detail levels can be produced from simple merges of 
regions from segmentations at finer detail levels. Therefore, the seg- 
mentations at finer levels are nested with respect to those at coarser 
levels. Hierarchical methods have the interesting property of pre- 
serving spatial and neighboring information among segmented re- 
gions. Here, we propose a hierarchical image segmentation in the 
framework of edge-weighted graphs, where the image is equipped 
with an adjacency graph and the cost of an edge is given by a dis- 
similarity between two points of the image. 

Any hierarchy can be represented with a minimum spanning 
tree. The first appearance of this tree in pattern recognition dates 
back to the seminal work of Zahn |T|. Lately, its use for image seg- 
mentation was introduced by Morris et al. |2| in 1986 and popu- 
larized in 2004 by Felzenswalb and Huttenlocher [3 1. However the 
region-merging method |3| does not provide a hierarchy. In HE), 
it was studied some optimality properties of hierarchical segmen- 
tations. Considering that, for a given image, one can tune the para- 
maters of the well-known method 1 3 1 for obtaining a correct segmen- 
tation of this image. We provide in this paper a hierachical version 
of this method that removes the need for parameter tuning. 

The algorithm of [ 3 1 is the following. First, a minimum span- 
ning tree (MST) is computed, and all the decisions are taken on this 
tree. For each edge linking two vertices x and y, following a non- 
decreasing order of their weights, the following steps are performed: 

(i) Find the region X that contains x. 

(ii) Find the region Y that contains y. 

(iii) Merge X and Y according to a certain criterion. 

The criterion for region-merging in | 3 1 measures the evidence 
for a boundary between two regions by comparing two quanti- 
ties: one based on intensity differences across the boundary, and 



the other based on intensity differences between neighboring pixel 
within each region. More precisely, in step (iii), in order to know 
whether two regions must be merged, two measures are consid- 
ered. The internal difference Int(X) of a region X is the highest 
weight of an edge linking two vertices of X in the MST. The differ- 
ence Diff(X, Y) between two neighboring regions X and Y is the 
smallest weight of an edge that links X ioY . Then, two regions X 
and Y are merged when: 

Diff{X,Y) < mm{Int(X) + ^-Jnt(Y) + A} (1) 

where k is a parameter allowing to prevent the merging of large re- 
gions (i.e., larger k force smaller regions to be merged). 

The merging criterion defined by Eq. |T} depends on the scale 
k at which the regions X and Y are observed. More precisely, let 
us consider the (observation) scale Sy{X) of X relative to Y as a 
measure based on the difference between X and Y, on the internal 
difference of X and on the size LX1 of X: 

S Y (X) = (Diff(X,Y) - Int(X)) x \X\. (2) 
Then, the scale S(X, Y) is simply defined as: 

S(X,Y) = max(SVpO,Sx(F)). (3) 

Thanks to this notion of a scale Eq. {T} can be written as: 

k>S(X,Y). (4) 

In other words, Eq.(|4| states that the neighboring regions X and Y 
merge when their scale is less than the threshold parameter k. 

Even if the image segmentation results obtained by the method 
proposed in |3 1 are interesting, the user faces two major issues: 

• first, the number of regions may increase when the parameter 
k increases. This should not be possible if k was a true scale 
of observation: indeed, it violates the causality principle of 
multi-scale analysis, that states in our case 1 6| that a contour 
present at a scale k± should be present at any scale k<z < ki . 
Such a behaviour is demonstrated on Fig.[T] 

• Second, even when the number of regions decreases, contours 
are not stable: they can move when the parameter k varies, 
violating a location principle. Such a situation is illustrated 
on Fig. [2] 

Given these two issues, the tunning of the parameters of (3) is a 
difficult task. 

Following 1 6 1, we believe that, in order for k to be a true scale- 
parameter, we have to satisfy both the causality principle and the 
location principle, which leads to work with hierarchy of segmenta- 
tions. Reference 1 7 1 is the first to propose an algorithm producing a 
hierarchy of segmentations based on \3 \. However, this method is an 




(a) Original (b) k = 7500 (8) (c) k = 9000 (14) 



Fig. 1. A real example illustrating the violation of the causality prin- 
ciple by 1 3 1: the number of regions (in parentheses) increases from 
8 to 14, instead of decreasing when the so-called "scale of observa- 
tion" increases. 




(a) Original image (b) k = 5 (c) k = 8 



Fig. 2. An example illustrating the violation of the location property 
by 0: the contours are unstable from one "scale" to another. 



iterative version of 1 3 1 that uses a threshold function, and requires a 
tunning of the threshold parameter. 

The main result of this paper is an efficient hierarchical image 
segmentation algorithm based on the dissimilarity measure of 0. 
Our algorithm has a computational cost similar to 0, but provides 
all scales of observations instead of only one segmentation level. 
As it is a hierarchy, the result of our algorithm satisfies both the 
locality principle and the causality principle. In particular, and in 
contrast with [3 1, the number of regions is decreasing when the scale 
parameter increases, and the contours do not move from one scale to 
another. 

Figure [3]illustrates the results obtained by applying our method 
to the same image of Fig. with segmentations at two different 
scales of observations, as well as a saliency map 1 8 4 5 1 (a map 
indicating the disparition level of contours and whose thresholds give 
the set of all segmentations). 




(a) Saliency map (b) k = 1000 (22) (c) k = 5000 (6) 



Fig. 3. A real example illustrating the saliency map of Fig. [TJa) 
computed with our approach. We display in (b) and (c) two image 
segmentations extracted from the hierarchy at scales 1000 and 5000, 
together with their number of regions (in parentheses). 

This work is organized as follows. In Section [2] we present our 
hierarchical method for color image segmentation. Some experi- 
mental results are given in Section [3] Finally, in Section [4] some 
conclusions are drawn and further works are discussed. 



2. AN EFFICIENT HIERARCHICAL GRAPH BASED 
IMAGE SEGMENTATION 

In this section, we describe our method to compute a hierarchy of 
partitions based on observation scales as defined by Eq. [3] Let us 
first recall some important notions for handling hierarchies |2||4][5). 

To every tree T spanning the set V of the image pixels, to every 
map w : E — »• N that weights the edges of T and to every threshold 
A G N, one may associate the partition V\ of V induced by the 
connected components of the graph made by V and the edges of 
weight below A. It is well known |2|[5) that for any two values Ai 
and A2 such that Ai > A2, the partitions V\ x and V\ 2 are nested 
and Vx x is coarser than Vx 2 - Hence, the set H w = (V™ | A G N) 
is a hierarchy of partitions induced by the weight map w. 

Our algorithm does not explicitly produce a hierarchy of parti- 
tions, but instead it produces a weight map L (scales of observations) 
from which the desired hierarchy H L can be infered. It starts from 
a minimum spanning tree T of the edge- weighted graph built from 
the image. In order to compute the scale L(e) associated with each 
edge of T, our method iteratively considers the edges of T in a non- 
decreasing order of their weights. For every edge e, the weight map 
L(e) is initialized to 00; then for each edge e linking two vertices x 
and y the following steps are performed: 

(i) Find the the region X of V™^ that contains x. 

(ii) Find the the region Y of P™( e ) that contains y. 

(iii) Compute the hierarchical observation scale L(e). 

At step (iii), the hierarchical scale S'y(X) of X relative to Y is 
needed to obtain the value L(e). Intuitively, S'y(X) is the lowest 
observation scale at which some sub-region of X, namely X*, will 
be merged to Y. More precisely, using an internal parameter v, this 
scale is computed as follows: 

(1) Initialize the value of v to 0. 

(2) Increment the value of v by 1 . 

(3) Find the the region X* of Vy that contains x. 

(4) Repeat steps 2 and 3 while Sy (X*) > v 

(5) S' Y (X) = v. 

With the appropriate changes, the same algorithm allows S'x (Y) to 
be computed. Then, the hierarchical scale L(e) is simply set to: 

L(e) = max{SV(X),S'xOn}. (5) 

Fig. [4] illustrates the result of our method on a pedagogical ex- 
ample. Starting from the graph of Fig. |4ja), our method produces 
the hierarchical observation scales depicted in Fig.Qb). As for the 
method of |3 |, our algorithm only considers the edges of the mini- 
mum spanning tree (see Fig.[4|c)). The whole hierarchy is depicted 
as a dendrogram in Fig.[4jd), whereas two levels of the hierarchy (at 
scales 2 and 9) are shown in Fig.[4]e) and (f). 

Let us illustrate the computation of a hierarchical obervation 
scale on the graph of Fig. [5ja). To this end, we consider the itera- 
tion of the algorithm at which the edge e linking B to G is analyzed. 
At this step, the edges of the MST of weight below w(e) = 10 
have been already processed. Therefore, the hierarchical observa- 
tion scale of these edges (depicted by continuous lines in the figure) 
is already known as shown in[5|b). The regions X and Y obtained at 
steps (i) and (ii) are set to {A, B,C,D, E} and {F, G, H, 1} respec- 
tively. Then, in order to find the value L(e) at step (iii), the partitions 
Vf for i = {2, 7, 13, 18} must be considered. We have: V2 = 
{{£, C}, {A, D}, {£}, {G}, {F}, {I, if}}, V 7 L = {{B, C}, 
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(a) Original graph 
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(d) Dendrogram 



Fig. 4. Example of hierarchical image segmentations. In contrast to 
example in Fig. [2] the contours are stable from a scale to another, 
providing a hierarchy. 




Fig. 6. An example of a hierarchical image segmentations of a 
synthetic image containinng three perceptually big regions. The 
saliency map of the image (a) is showned in (b). The number of 
regions of the segmented images is written under each figure. 




(d) 



(a) 



(b) 



Fig. 5. Example for computing the hierarchical scale for an edge- 
weighted graph. For this example, we suppose that all scales for the 
regions X and Y are already computed, and we will calculate the 
hierarchical scale for the edge connectinge B and G. 



Fig. 7. Top row: some images of the Berkeley database | 9 |. Middle 
row: saliency maps of these images. The numbers of scales of these 
hierarchies are (a) 240, (b) 429, (c) 405 and (d) 443.Bottom row: ac- 
cording to our subjective judgment, the best segmentations extracted 
from the hierarchies. The numbers of regions are (a) 3, (b) 16, (c) 6 
and (d) 18. 



{A, D, E}, {F}, {G, H, /}}, Vts = {{#, C}, R A E}, {F, G, 
H, I}} and Vts = {{B, C}, {A, D, E}, {F, G, H, I}}. By the ap- 
plication of steps (1-5), the value S'y (X) is found to be 18 since 18 
is the first value below the observation scale of the region contain- 
ing B relatively to Y. The same process is made for S' x{Y), but 
the regions are {G}, {G, H, 1} and {G, H, /, F}. Moreover, the 
observation scale is 12 since 12 is the first value below the obser- 
vation scale of the region containing G relatively to X. Finally, the 
observation scale of X and Y is 18. 

To efficiently implement our method, we use some data struc- 
tures similar to the ones proposed in (5); in particular, the manage- 
ment of the collection of partitions are due to Tarjan's union find. 
Furthermore, we made some algorithmic optimizations to speed up 
the computations of the observation scales. In order to illustrate an 
example of computation time, we implemented all our algorithm 
in C++ on a standard single CPU computer under windows Vista, 
we run it in a Intel Core 2 Duo, 4GB. For the image illustrated in 
Fig.[TJa) (with size 321x481), the hierarchy is computed in 2.7 sec- 
onds, and the method proposed in |3 1 spent 1.3 seconds . 

3. EXPERIMENTAL RESULTS 

A major difficulty of experiments is the design of an adequate edge- 
cost, well adapted to the content to be segmented. A practical so- 
lution is to use some dissimilarity functions, and many different ap- 
proaches are used in the litterature. In this work, the underlying 



graph is the one induced by the 4-adjacency relation, and the edges 
are weighted by a simple color gradient computed by an Euclidean 
distance in the RGB space. 

In Fig. [6] we present some results on an artifical image contain- 
ing three perceptually big regions. On this example, one can easily 
verify the hierarchical property of our method by looking at the seg- 
mentations at scales resp. 1000, 2000, 5000, 140000 and 224000 
(resp. Fig.|6jc), (d), (e), (f) and (g)). Since the resulting segmenta- 
tions are nested, the whole hierarchy can be presented in a saliency 
map (see Fig. [6jb)). 

Fig.|7]illustrates the performance of our method on some images 
of the Berkeley's database |9 |. Note that, as in 1 3 1, an area filtering 
is applied to eliminate small regions (smaller than 500 pixels). 

Comparison of the results of our algorithm with the ones of 
are difficult, since the tuning of the parameters of is critical and 
since we produce a whole hierarchy of segmentations. We made 
three experiments. First, we try to set the correct parameter for (3), 
i.e. the parameter that produces the best (subjective) visual result 
(Fig. [8}. We can compare this result with on the one hand, the "best" 
segmentation extracted from our hierarchy in Fig. [7] and on the other 
hand, with a segmentation from our hierarchy containing the same 
number of regions as (3) (Fig. [8}. In a second experiments, we fixed 
the number of regions to 15 for all images, and tune the parame- 
ter of to obtain this number of regions. We can compare these 
segmentations with our own results on Fig. [9] The last experiments 
is designed to assess the robustness to random impulse noise, see 

Fig.rroi 



(a) 16 



(b) 52 




(c) 26 



(d) 18 



Fig. 8. Comparison between |3] and our approach. For each pair 
of images, the right image shows the best result (according to our 
judgment and our experiments) from |3 1 and the left image shows a 
segmentation extracted from our hierarchical result, with the same 
number of regions. 




Fig. 10. Examples of segmentations for images corrupted by a ran- 
dom salt noise. The corrupted images (at different levels - 70% and 
90%) are shown on the first column. The results of our method and 
1 3 ] are illustrated in the second and third columns, respectively. 



4. CONCLUSIONS 
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Fig. 9. Examples of image segmentation where the number of re- 
gions has been set to 15. For each pair of images, the left one shows 
a segmentation extracted from our hierarchy, with the desired num- 
ber of regions; and the right one shows the result obtained with (3j 
by varying the parameter k until the desired number of regions is 
found. 



This paper proposes an efficient hierarchical segmentation method 
based on the observation scales of 0. In contrast to | 3 1, our method 
produces the complete set of the segmentations at every scales, and 
satisfies both the causality and location principle defined by | 6]. An 
important practical consequence of these properties is to ease the 
selection of a scale level adapted to a particular task. We visually 
assessed our method on some real images by comparing our seg- 
mentations to those of 1 3 1. Even if more (quantitative) tests (such as 
the ones proposed by 1 10 1) are needed for drawing definitive conclu- 
sions, the produced segmentations are promizing, in particular w.r.t. 
robustness. As future work, we will investigate using more informa- 
tion into the definition of observation scale as well as learning which 
information is pertinent for a given practical task. 
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